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ABSTRACT 

An  algorithm  is  described  that  determines  a stationary 
point  of  a quadratic  minimization  problem  in  a finite  number 
of  steps.  This  finite  termination  property  is  based  on  the  use 
of  conjugate  directions.  The  main  feature  of  the  algorithm  is 
a new  update  procedure  which  preserves  conjugate  directions 
if  the  set  of  active  constraints  changes. 
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* 


A method  of  conjugate  directions  is  presented  for  the  solution  of  quad- 
ratic minimization  problems  with  linear  inequality  constraints.  The  algorithm 
terminates  after  a finite  number  of  steps  with  a stationary  point.  It  is  a modi- 
fication of  methods  of  conjugate  directions  for  general  nonlinear  objective 
functions  described  in  [1]  and  [3], 

With  each  point  x determined  by  the  algorithm  an  (n,n)-matrix  is 
associated,  where  n is  the  number  of  variables.  If  q < n constraints  are 
active  at  x , then  n - q columns  of  this  matrix  are  conjugate  and  orthogonal 
to  the  gradients  of  all  constraints  active  at  x . This  property  allows  an  easy 
construction  of  search  directions  which  are  either  Newton  directions  or  are 
conjugate  to  certain  previous  search  directions.  Combined  with  an  appro- 
priate policy  for  dropping  active  constraints  this  choice  of  search  directions 
results  in  the  finiteness  of  the  algorithm. 

A critical  feature  of  the  method  to  be  presented  is  the  procedure  used 

A 

to  update  the  matrix  associated  with  x . If  at  the  next  point  x,  constructed 
by  the  algorithm,  no  new  constraint  becomes  active  this  matrix  is  updated  in 
the  same  way  as  the  basis  matrix  is  updated  in  the  simplex-method.  If  how- 

A 

ever,  a new  constraint  becomes  active  at  x,  the  normal  update  procedure 
results  in  the  loss  of  the  conjugate  directions.  Therefore,  a new  update 
formula  is  developed  which  allows  the  preservation  of  conjugate  directions  in 


a simple  and  computationally  efficient  way. 

Sponsored  by  the  United  States  Army  under  Contract  No.  DAAG29 -7  5-C -0024. 


2.  General  description  of  the  algorithm 


We  consider  the  following  quadratic  minimization  problem;  Minimize 
Q(x)  = c'x  + \ x'Cx 
subject  to  the  constraints 
Ax  < b , 

where  c,  x « En,  be  Em,  C is  a symmetric  (n , n) -matrix  and  A is  an  (m,n(- 
matrix. 

If  x*  is  a local  or  an  optimal  solution  to  this  problem  it  follows  from 
the  Kuhn-Tucker-Theorem  (see  e.g  [2])  that  there  is  a vector  u*  t E™  such 


that 

(2.1) 

c + Cx*  = A'u* 

(2.2) 

u*'(Ax*  - b)  = 0 , 

(2.  3) 

Ax*  < b . 

Any  point  satisfying  the  conditions  (2.1)  - (2.  3)  is  called  a stationary 
point.  If  C is  positive  semi-definite  then  Q(x)  is  convex  and  every  stationary 
point  is  an  optimal  solution  to  the  given  problem. 

Throughout  the  paper  we  assume  that  for  any  x with  Ax  < b the  gra- 
dients of  all  constraints,  active  at  x , are  linearly  independent  and  that  the 
set  { x | Ax  < b and  Q(x)  < Q(x^)}  is  bounded. 

Let  Xj  with  Ax^  < b be  a point  determined  by  the  algorithm.  For  ease 
of  notation  we  assume  that 

a!x.  = (b).  , i = 1,...  ,q 

and 

ajx.  < (b).,  1 = q+1, . . . , m , 


-2- 


where  a!,.  . . ,a'  denote  the  rows  of  A . 

1 m 

If  q < n , set 

T.  = {x|  alx  = 0 , i = . 

We  first  assume  that  C is  positive  definite.  Then  we  can  construct  a 
set  of  n-q  vectors 

(2.  4)  c t T , i = q+1, . . . ,n 

which  form  a basis  of  the  subspace  T.  and  are  conjugate  with  respect  to  C , 
i.e.  , have  the  property 
(2.  5) 

With 

(2.6) 


cl.Cc..  = 0,  c! . C c.  .=  0. . , i * k,  i,  k = q+1, 
ij  k]  ’ ij  u U 


,n. 


d;  • <» •->  Vmc  Vw e-Co-’ 


) 1 q 

it  follows  from  (2.  4)  and  (2.  5)  that 


nj  nj 


D 


-1 


J 


= <V  •■■.cnJ) 


exists  and  has  the  vectors  (2.4)  as  its  last  n-q  columns. 

In  the  algorithm  this  matrix  ^ and  the  numbers 

G.  . = [cl.Cc..]"1,  i = q+1, . . . , n 
t]  1 ij  UJ 

are  associated  with  x.  . 

Given  the  point  x,  the  algorithm  determines  a new  point 

x.  . = x - <r  .s . , 

J+l  J J j 

where  s « En  is  the  search  direction  and  <r . t E is  the  step  size. 

) ) 

In  order  to  motivate  the  choice  of  s^  (see  Step  1 of  the  algorithm)  we 

write  the  gradient  of  Q(x)  at  x.  in  the  form 

q ] m 

(2.7)  c + C x.  = V .a,  + X k. . C c. . . 

J i=l  1J  1 i=q+l  13 

If  \ . = 0 , i = q+1, . . . ,n,  i.  e.  , if  the  orthogonal  projection  of  c + Cx. 
ij  j 


M 


In 


.3- 


r 


1 


L 


onto  is  zero,  then  x^  is  said  to  be  a quasi-stationary  point. 


First  we  assume  that  x is  not  a quasi-stationary  point.  If  we  set 
n J 


(2.8) 

then 


= y 


s = / (0  . c1  g )c 

J i=q+l  iJ  iJ  J 1J 


x - s 
J ) 


with  g = c + Cx 
J J 


is  a quasi-stationary  point.  Indeed,  it  follows  from  (2.4)  and  (2.5)  that  the 

vectors  c . c . form  a basis  for  T..  Furthermore,  for  every  k< 

q+1 , ) ’ nj  j ’ 


iq+i, . • • ,n}  , 


'kj 


J J 


c!  (c 

+ Cx. 

- Cs.) 

kj 

) 

J 

n 

c!  .g. 

kl  J 

' Ck) 

C[v  (G.  ,c!  g.)c. .] 
i=q+l  1J  J 

c.1  .g. 
krj 

" °kj 

c cw  Vj  = 0 

Assuming  that  A(x.  - s .)  < b , we  can  choose  <j^  = l and  x.+^  is  a quasi- 
stationary  point  which  in  the  case  of  a convex  objective  function  is  an  optimal 
solution  to  the  problem 


min(Q(x)  | a|x 


(b). , i = 1, . . . ,q,  a!x  < (b).,  i = q+1, . . . ,m  }. 


Since  the  same  constraints  are  active  at  x.  and  at  x.  , we  have  T,  = T , 

J J+l  ) j+1 

and  can,  therefore,  choose  D ^ = D . * (Step  3 of  the  algorithm). 

Next  we  assume  that  x.  is  a quasi-stationary  point  . Multiplication 
of (2.7)  with  c , i = 1, . . . ,q  , gives 

c! . g.  = . . 

ij  J i) 

Thus,  either  x^  is  a stationary  point  or  there  is  at  least  one  positive  number 


eg. 
U J 


Let 


cq.  = max{cj  g | i = 1, . . . ,q}  > 0 . 


If  we  set  s.  = c . , it  follows  from  (2.  6)  and  the  definition  of  the  inverse  matrix 

j qj  ' 
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r 


that  s is  orthogonal  to  Cc  , i=q+l,...,n.  Therefore,  for  all  <t  , 
J ij 


V Q(x  - o-  s ) is  orthogonal  to  c , 

J J q+1 , J 

If  o-  is  the  optimal  step  size,  i.e.  , 


nj 


Q(x  - cr.  s ) = min[Q(x . - a s .)  I <r  > 0 } , 

J J j J J 

then  V Q(x  - a . s ) is  orthogonal  to  s.=c  c . ....  ,c  , . Assuming  that 

J J J J qj  q+l,J  nj 

A(x.  - <r.  s . ) < b , we  can  choose  cr . = if . . Then  x , is  again  a quasi-stationary 
J J J — J J J+l 

point.  Since 

Tj+1  = {x  | ajx  = 0 , i = 1, . . . , q -1 } 

we  obtain  the  matrix  D'  , from  D!  byreplacing  a with  8 .Cc  ..  Then 

J+l  J q qj  qj 

°j  + l r <Cl,j+l’---’Cn,j+l)  ’ 


where  (see  Step  4 of  the  algorithm) 


c.  . , = c. . 
i , J+l  U 


c! . C c . 

_iL ELL  c 

c'  . C c . qj 

qj  qj 


i = l,..  . , q -1 


c.  . , . = c. . , 

i,J+l  U 


l = q,.  . . ,n. 


Thus  the  vectors  c . .....  ,c  . are  again  conjugate  with  respect  to  C . 
q , J+l  nj 

Therefore,  if  no  new  constraint  becomes  active  at  x j,  then  x^  is 
a quasi-stationary  point  and  it  is  easy  to  obtain  a set  of  conjugate  directions 


forming  a basis  for  T If  a new  constraint  becomes  active  at  x.+^,  the 
situation  is  completely  different.  To  be  specific  assume  that  s^  is  given  by 
(2  8)  and  that  a is  the  gradient  of  the  new  active  constraint.  In  order  to 

q+i 

guarantee  that  s.+^  is  a feasible  search  direction,  the  vectors  a^,...,a^+^ 

have  to  be  among  the  columns  of  , i.e.  , we  have  to  replace  one  of  the 

vectors  Cc  C c . with  a Assume 

q+1,  J nj  q+1 

D'.  , = (a  , . . . , a .,0  , C c „ .,...,6  .C  c ). 

J+l  1 q+1  q+2,  J q+2,  j nj  nj 
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Then  the  columns  of  D ^ are  given  by 

c 

c _ 

q+1 , j+l  c',  .a 

q+i,j  q+i 

c! . a 

u q+i  . 

c.  . . = c. . - — - — — c . i=l,.  . . , n , 1 * q . 

l>’+1  11  c<,ti,iVi  qtl’’ 

Thus  the  vectors  c _ c are  conjugate  if  and  only  if  c! , a . = 0, 

q+2,  j+l’  ’ n,j+l  ij  q+l  ’ 

i = q+2, . . . , n , i.  e.  , if 

Vi'  spaniiV---’V  c Vi,j’  ■ 

A similar  argument  applies  to  the  case  where  s.  = c . . 

J qj 

Of  course  it  is  possible  to  construct  a new  set  of  conjugate  directions 
which  form  a basis  of  T j.  However,  if  q is  much  smaller  than  n , then 
this  is  a time  consuming  procedure  and  the  resulting  algorithm  cannot  be  ex- 
pected to  be  very  efficient.  Therefore,  it  is  important  to  develop  an  update 
formula  for  D.  * which  allows  us  to  easily  "transfer"  conjugate  directions 

from  the  subspace  T.  into  the  subspace  T.  , in  the  case  where  a new  con- 

J J+l 

straint  is  active  at  x . Such  a procedure  can  be  based  on  the  following 
lemma. 

Lemma  1 


Let  r«  {l,...,n}  and  let  G be  a symmetric  ( n, n) -matrix  such  that 


there  are  vectors 


Pj  > • • • > Pr  ( E 


p!  Gp,  = 0 , 1 < i < k < r 

1 K 


p!  G p.  > 0 , i = l, . . . ,r  . 
ii 
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Let  a t E'  be  such  that 


r p|a 

p -j  p1  G p.  pi 
i = l 1 l 


Let  ft  (1, . . . ,r}  . If  p' G p p'a  - (p'  a)  = 0 , set 


q = P{  , i = 1, . . . ,r  , i * t , 


otherwise  set 


P1  a(l  - t pj  a) 

p.  - —i- — ^ P ■ t P|  a pf  , i - I > • • • > r,  i * l , 


where  t is  a solution  of  the  equation 


G 

P(  P'a  - 

(p;a>2: 

) + 

2 t p'  a - 1 = 0 , 
f 

i) 

a'qi 

= 

0 

i = 

1*. 

• • »r  , 

i * 

t 

ii) 

q;G 

qk 

= 0 , 

i,lu 

Ill 

, . . . ,r}  - i*  ] , 

i * 

k 

iii) 

q;G 

qi 

= piG 

Pt  > o. 

i 

- 1 ) • • • > r , 

i * 

t 

iv) 

span 

IP* 

V 1 = 

1, . . . ,r 

'*  i 

* f } = span{Pj, . 

• • > 

Pr> 

Proof. 


Suppose  p^  G pf  p’a  - (p'  a)  = 0 . Since 

r (P!a)2  2 

PlGP.P'a^PiGp,  V — — + (p;a) 

1=1  1 1 


it  follows  that  p!  a = 0,  i = 1,.  . . ,r  , i * £ . Hence  the  first  3 statements  of 

Z 

the  lemma  are  true.  If  p^  G p(  p'a  - (p'  a)  * 0,  then  the  quadratic  equation 
defined  in  the  lemma  has  two  distinct  real  solutions.  Set 
(2.9)  d = G p,  d.  = G p.,  p.  = p!a  , i=l,...,r 

and  observe  that  p'a  = p'd  and 


p’d.  = p'  G p,  = d'p.  = a'p.  = p.  , i=l,...,r. 


-7- 


fVO  - tPj) 

a,qi  = d'qi  = d'(pi- p-‘Pi  pr 


= d'p.  - p.  + t p.pf  - t p.  d'pf  =0  for  i = 1, . . . ,r,  i * t . 
Furthermore,  for  i,kt  {1 , . . . , r}  - {I  } , 


qiGqk 


p <i  - t p ) p (i  - tp  ) 

(Pr  - d^-P-'Wn  d^~d  -^k  V 


= Pldk 


p.(l-tpf)  Ml-tp) 

d'p  pk  + dTp  Pk(V  + 1 PiPk  Pfd^ 


PiPk  2 Z 

= p;dk  +~d^ti  (d,pd;pf  -pp  + 2pf  t-i] 

= p; G pk  • 

This  proves  parts  ii)  and  iii)  of  the  lemma.  In  order  to  prove  the  last  state- 
ment of  the  lemma,  it  suffices  to  show  that  the  vectors  p,  q, , . . . , q , q 

’ 1 ’ -1’  +1’ 

. . . , are  linearly  independent.  Let 

Xp  + X^  + ...  + VlVl  + Xf+1qf+1  + ...  + Xfqr  = 0 . 

Skalar  multiplication  with  Gp  and  Gp,  , respectively,  gives 
Xp1  G p - 0 and  X^pj,  G p^  = 0 


from  which  it  follows  that  X = X = . . . X = X 

1 t-i  t+ 1 


X = 0 . 
r 


In  order  to  show  how  this  lemma  can  be  used  to  construct  a basis  of 

conjugate  directions  for  T. ,,  we  assume  that  a , is  the  gradient  of  the  con- 

) +1  q+1 

straint  that  becomes  active  at  x.  Setting  G = C,  a = a , , r = n - a.  f=l 

j+1  q+1 

and  p.  = c . , i = 1, . . . ,r,  we  conclude  from  the  lemma  that  the  vectors 

l q+r , J 


(2.10) 


Cq+i , j+1  qi  ’ 


i = 2,. . . ,r  , 


form  a basis  of  conjugate  directions  for 


Tj+1  = |X  I aix  = °>  i = - • • >9+1}  • 


A 
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If  is  given  by  (2.B),  then  the  first  q+1  constraints  are  active  at 


x . and  D'  . 
J+1  j+1 


,al dqtl’  V,]MCV,i.l""'en.l*lCO' 


n , j+1  n,  j+1 


Therefore,  the  remaining  columns  of  D.+j  are  given  by  (see  Step  5 of 
the  algorithm  and  Lemma  2). 


q+i,]+i  p'aqf] 

C[j  °q+l 

c.  , = c - — p — - p , 

i,l+l  U pa 

q+1 


i = 1, . . . ,q  . 


If  on  the  other  hand  s,  is  equal  to  c , the  q-th  constraint  is  not 

1 qi 

active  at  x j.  Thus  we  need  a basis  of  conjugate  directions  for 

Tj+1  = txia|x  = 0,  aq+]x  = 0,  i = l,...,q-l}. 

In  order  to  apply  the  lemma  we  observe  that  by  (2.  6)  the  vectors 

c ....  ,c  . are  conjugate  with  respect  to  C and  are  elements  of  the  sub- 
qi  nj 

space 

{x!  a|x  = 0 , i = 1, . . . ,q-l}  . 

With  G = C,a=a  ,,  r = n-  q+1,  t = 1 and  p.  = c . . . , i = . . . , r it 
q+1  l q-l+i,  j ’ 

follows  from  Lemma  1 that  the  vectors 

Cq-l+i, j+1  = qi  ’ 1 = 2,1  • • ,r 

❖ 

form  a basis  of  conjugate  directions  for  T^+^.  Since 

D! ' = (a  . . ,a  ,a  . ,6  , . . , ,C  c .......  ,0  . ,C  c 

1+1  1 q-1  q+1  q+1,  J+1  q+1,  j+1  ’ n,j+l  n,j+r 

the  remaining  columns  of  D ^ are  given  by 


"q , j+1  P'a, 


c.  . , = c. . 

i,]+l  u 


c . a , 
U q+1 


i = l,.  • • ,q-i 


So  far  we  have  assumed  that  C is  positive  definite.  If  this  is  not  the 


case  and  if  s = c . it  can  happen  that  s'  C s.  <0  since  then  Q(x.  -as.) 

1 qj  J J - 11 


-9- 


t 


is  linear  or  strictly  concave  no  optimal  step  size  exists  and  a new  constraint 

becomes  active  at  x Since  C c < 0 we  cannot  apply  Lemma  1 

directly.  However,  we  observe  that  the  vectors  (2.10)  obtain  by  applying 

Lemma  1 to  c , , . . . , c together  with 

q+l,j  nj 

c'  a , 
qj  q+1 

c . . = c - — ~ — - — p 

q+1 , j+1  qj  p'  aq  + 1 

if 

form  a basis  of  conjugate  directions  for  T . Indeed,  since  all  tnese  vectors 

* 

are  in  T,+j  and  since  the  q,  are  conjugate  it  suffices  to  show  that 
q!  Cc  =q!Cp=0,  i = 2, . . . , r . 

i qj  i ’ 

Observing  that  T.+1  C (x  : a!x  - 0 , i=l,...,q)  we  conclude  from  (2.  6)  and 
the  definition  of  D.  1 that  q'  C c . = 0 for  all  q £ T.  Furthermore,  since 

j qj  j+i  ’ 

d'q.  = a'q.  it  follows  from  (2.9)  and  part  i)  of  the  lemma  that  q!  C p = 0 for 
i = 2, . . . ,r  . 


Since  p'  C c = 0 , we  have 

qj 

( C . a )2 

CqH  ,i«  C °q«  , J+1  = CQi  C Cqj  + P'a  P'  C P ‘ 

q+i 


(2.12) 


If  this  number  is  positive,  then  C is  positive  definite  on  the  subspace  T . 

As  in  the  case  of  a positive  definite  matrix  C , the  matrix  D is  given  by 

(2.11).  However,  since  c , . , is  constructed  differently  p and  c 

q+1,  J+1  q+1,  j+1 

are  in  general  not  conjugate.  This  results  in  a more  complicated  formula  for 
the  vectors  c.  c (See  Sten  7d  of  the  algorithm).  If  (2.12)  is 

IjJ+l  M j J + 1 

not  positive  we  choose 


Dj+1  fal’ ' ■ ’ ’ iq-l’aq+l,aq,8q+2,  j+lC  °q+2,  j+1’  ‘ " *6n,  j+1  C °n,  j+l'- 


Then  (see  Step  7c  of  the  algorithm) 

o! . a , 

c.  = c.  . - — ~ — — — p,  i = 2, . . . , r . 

.,J+I  1)  P'aqt| 

Note  that  a is  the  (q+l)-th  column  of  D1.  , even  though  the  q-th  con- 
q J+l 

straint  is  not  active  at  x.  In  order  to  correct  this  situation  we  choose  s . 

J+l  J+l 

parallel  to  c ....  Since  Q(x  , - <j  s.  ,)  is  concave  no  optimal  step 
q+l  > j+l  j+l  j+l 

size  exists  and  an  additional  constraint  is  active  at  x , Thus  after  a 

j + 2 

finite  number  of  steps  we  have  either  an  extreme  point  (Step  7a  of  the  algo- 
rithm) or  a positive  number  (2.12).  (Step  7d  of  the  algorithm).  In  either  case 

a is  removed  from  the  corresponding  matrix  D! 
q J+v 

3 . Detailed  statement  of  the  algorithm. 

It  is  assumed  that  the  algorithm  starts  with  a feasible  extreme  point  of 

the  set  {xfAx  < b).  If  such  a point  is  not  available  it  can  be  determined  by 

solving  a linear  programming  problem. 

We  describe  now  a general  cycle  of  the  algorithm.  At  the  beginning 

of  the  j-th  cycle  the  following  data  are  available;  a feasible  x^  , the 

gradient  g = c + Cx  of  CM*)  at  x , the  numbers  (3.=0  or  1,6  =0 
J J J J J 

or  1 , the  matrix  D. 1 = (c,  ......  c .)  and  the  set  J(x. ) = i a ., . . . ,a  } . 

J lj  nj  j lj  nj 

The  or  ,'s  are  nonnegative  integers.  If  = 0 , then  c.  . is  a conjugate 

direction.  If  a . > 0 , then  the  constraint  with  subscript  a.  is  active  at 
lj  i) 

x and  the  i-th  column  of  D is  equal  to  the  gradient  of  this  constraint. 

J J 

(3.  = 1 if  and  only  if  a new  constraint  became  active  at  x^.  6^  is  equal  to 

1 if  and  only  if  s.  = c(  . is  a search  direction  along  which  Q(x)  is  not 
strictly  convex.  Step  7 of  the  algorithm  is  used  to  deal  with  this  situation. 
Finally,  = 0 or  -1  is  determined  in  Step  I of  the  algorithm.  = 0 if 
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Jnd  only  if  the  search  direction  s.  is  such  that  ail 

x will  also  be  active  at  x 
J j+r 

Step  1. 

a)  Compute 


constraints  active  at 


;9.  for  all  i with  a = 0 
11  1 ij 


c.  g - 0 for  all  i with  a =0 
1 1 ij 

go  to  Step  2b,  otherwise  set 
and  go  to  Step  Z. 


b)  Compute  ( such  that 


Cijgj  > Cijgj  ^or  4 with  a.  . > 0 


ff  Cfjgj  ~ st°P:  otherwise  set 


J fi  ’ yj 


Y = -1 


and  go  to  Step  2. 


Step  2. 


Compute 


i = 1, . . . ,m 


a!s.  > 0 , 
i J ~ 


i = 1, . . . , m 


(T  = 00 

J 


otherwise  compute  k such  that 


X!  ~ (b)k  ^ alXj  - 
clk s,  - a;  s 
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and  set 


Set 


<r . = 

J 


r 


^siCs, 


alV  Ib>k 

a'  s. 

k J 


if 

if 

if 


\ . 0 


s'.  C s.  < 0 and  v = -1 
J J - Yj 


s'.  C s . >0  and  v = -l 
J J Y) 


Set 


<r  . = min  j a 
J 1 j 


x = x -a\s., 
J+1  J J ) 


V = ° + CXJ+1 


j+1 


1 

0 


if 

if 


a < if. 

J ~ J 

* ^ * 

a . > cr  . 

J J 


If  6.  = 1 go  to  Step  7a,  otherwise  do  the  following.  If  = ( 

go  to  Step  3.  If  p . = 0 and  y . = -1,  go  to  Step  4.  If  3 = 1 

J j+1 

go  to  Step  5a.  If  = i and  -y.  = -I,  go  to  Step  6. 

SteP  3.  (p.+1  = "Y  = 0,  no  change  in  set  of  active  constraints) 

Set 

Dj’ii = Dj_1  > j(xj+i> = j(xj)  and 

0 . . . . = 6 . . for  all  i with  a . = 0 . 

i,J+l  i)  i ) 

Replace  j with  j+1  and  go  to  Step  lb  . 

Step  4.  (P^  = 0 , \ - 1 "dropping  a constraint") 

Set 


c.  ...  = c., 

i,  J+1  i] 


for  all  i with  a , = 0 
ij 


Cf , j+1  = Cfj 


and 

and 
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c1j(9j  ; lV 


i.w  'M'  -vi’  n 

\,n  * *h  ' 1 = ‘-----n-  lil' 

= 0 


for  all  i # I with  a. . > 0 

U 


0 . , =0.. 

i,j+l  ij 

0 , = [s’.  C s. 

i , 3+1  1 1 

-1 


for  all  i with  a, . - 0 
i) 

,-l 


Dj+1  _ (cl,j+l'*",Cnfj+l)’  J(Xj+l)  " |Ql,j+l,-",an,i+l 
Replace  j with  j+1  and  go  to  Step  lb. 


Step  5.  ((3  j = 1,  \ . = 0 , "adding  a constraint1 


a) 


Choose  any  It  {1, . ..,  n } such  that  . = ° and  compute 


p = )'  (G. . c! . a.  )c. . , 

^ ' i]  iJ  k'  U 

a . = 0 
11 

Set  a . - k and  go  to  Step  5b)  , 

t J 

b)  Compute  wj  = Pj  a^  . If  j3)^  = 0 ’ set 

c = c for  all  i with  a. . = 0 
i,  j+1  i] 

and  go  to  Step  5c),  otherwise  compute 


t.  = 
J 


-enc;iak  + ^e(]“i 
“j  - Wk’2 


set 


c.  ...  = C,. 

1,1+1  11 


c . a 

H k 


<1-,i°ljak)pr  Wk10*) 


for  all  i with  o.  . = 0 and  go  to  Step  5c) 
c)  Set 


'<  J+1 


c.  ...  = c. . - 

i,]+l  i) 


c . a 

il  k 


p.  for  all  i * f with  a. . > 0 
oj  , j 11 
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i 1 > . . . , n , 


i * ( 


a —Q 

l,j+l  1J 

°i  > j+i  k 

°i,j+i  ~ °ij 
-1 


for  all  i with  a , = 0 
i,  J+l 


°j+l  " (cl,j+l,,*,'Cn,j+l)  ' J(Xj)  = lQl,j+l"--,an,j+l}- 
Replace  j with  j+l  and  go  to  Step  la  . 


Step  6.  ( =1,  y = -1  "adding  and  dropping  a constraint") 

If  s'.Cs.  < 0 , go  to  Step  7a,  otherwise  set  6 =0  and 

J J - * j+l 

0f  . = (sj  C s^)  \ Compute 

P.  = Y,  (e..c'a,)c..  + (0,  ,c'  a. )c. . 
j ^ ' ij  U k ij  f j tj  k'  tj 

ot  , =U 
IJ 

and  go  to  Step  5b). 

Step  7 . 

a)  If  q_  = 0 for  at  least  one  i go  to  Step  7b),  otherwise  compute 

_ _Cli_ 


f,j+i  c;.ak 

c!  tau 

ij  k 

c.  • , i = c. . - — r c„ . , 

i, J+l  U c .a.  ( ] 


i = 1, . . . ,n,  i * f . 


ij  k 


Set 


Vi  = 1 ■ Vi  = 0 


a.  . . = a 

i,  J+l  iJ 


i - 1, . . . , n,  i#f  , a ^ - k 

-1 


J(Xj+l)  ~ {al,j+l’- *•  ,an,j+l}’  °j+l  = (Cl,j+l’---’Cn,j+l'- 


Replace  j with  j+l  and  go  to  Step  lb. 

b)  Choose  any  r e {1, . . . , n } such  that  a = 0 , and  compute 


P.  = Yj  ( 6.  .c!  .a.  )c, . , 
J a“0  U U k ij 

ij 


<*>.  = pa. 

J J k 


and 
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If  UJ.  - 0 ,(c'  .a,  ) = 0 , set 

1 rj  rj  k 

c.  = c.  . , for  all  i #r  with  a - 0 
i,l+l  i,l  ij 


and  go  to  Step  7c,  otherwise  compute 

-G  .c'  .a  + \l  Q . w. 
t _ h l)  k ( j ) 

j u).  - 0 ,(c'  . a.  ) 

J ly  l)  k' 


c!  .a 

ci  i+1  = cii  * — “ (1  - t.c'.a  )p.  - (t.cl.a  )c  . 

*■> J+1  U <*>  j rj  k j j ij  k rj 

for  all  i + r with  q,  . = 0 and  go  to  Step  7c  . 
ij 

c)  Compute 

Cf  ak 

°m«  = cu  ond  ci,wConw 

If  ci  j+|  C > 0 > set  = 0 and  go  to  Step  7d),  otherwise  set 

6.+j  = 1 compute 

o' . a. 
rj  k 

c.  ij.i  = ; P for  all  i l with  <*  . > 0 

1,1+1  U w.  J ij 

and  go  to  Step  7e) 

d)  Set 


°M+1  ' tcf,j+i  C cf  j+l1 


p.  = 0 . . [c'  C c. . 

U / , j+ll  l , j+1  ij 


o!  .a 

i j k 

c.  , . = Ci  - p.  - p.  ,c.  . . for  all  i + l with  a > 0 

1,1+1  l)  Wj  ) Klj  t,j+ 1 ij 

and  go  to  Step  7 e . 
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e)  Set 

■Vi  1 

’ Vi 

= -1 

°i, j+1 

V 

i = l,...,n,  i * f , i * r 

“rj+l 

= k 

“*,J+1 

a,  . 
t J 

0 

if  6.  . = 1 
J+1 

if  6,x.  = 0 . 
J+  1 

Set 

°i, j+1 

= G. 
1J 

for  all  i with  a.  = 0 

i,J+  1 

J<V 

= <°1,J+1 

••  ,0n,j+l*’  Dj+  1 = (C1,  j+1  ’* 

If  Vi = 0 ' 

replace 

j with  j+  1 and  go  to  Step  la.  If  6.+  ^ = 1 

cM+l 

if  c ' . . , Q . , , > 0 

*, J+1  J+1  - 

” . . 1 a J. 

< ,3+1 

Replace  j with  j+ 1 and  to  to  Step  2. 


Cf,j+1  gj+l  < ° • 


Remark. 


It  follows  immediately  from  Step  2,  that  every  is  feasible.  Further- 
more, if  is  determined  by  Step  1,  then  g’s^  > 0 and  Q(xj+j)  < Q(xj) 

unless  tr  =0.  This  could  happen  if  several  new  constraints  became  active 

i 

at  x.  since  only  one  is  used  in  the  update  procedure  for  D._j.  If  s.  is 

computed  in  Step  7 of  the  algorithm,  then  Q(x.  - tr  s^)  is  concave  and 

q's  > 0.  Thus  we  have  always  Q(x.,  ,)  < Q(x,). 
j j - J+  A — J 


t 


4.  Termination  properties. 

In  this  section  we  prove  that  the  algorithm  terminates  after  a finite 
number  of  steps  with  a stationary  point.  The  following  lemma  establishes 
the  properties  of  the  matrix  D.  * on  which  this  result  is  based. 


Lemma  Z. 


Let  D.  (cij’  • • • ’cnj)’  J(Xj)  " ^alj»  • ' • »ani  } and  6^  be  determir 


by  the  algorithm.  Set 


T j - {x  | a'  x = 0 for  all  i with  a >01 
J "ij  ij 

T.  = span{c.  . | all  i with  a =0} 

c1j  c ci, = "ij1  > 0 - cIj c ck,  = 0 forail 

with  a. . = a = 0 . 

U kj  ' 

i.e.  the  vectors  c_,  for  all  i with  a-  = 0 , form  a basis  of 
gate  directions  for  T 

j 


conju- 


c|cj  C Cij  - 0 f°r  all  k with  > 0 and  all  i with 
a.  . = 0 . 

aa  Cii  = 1 > a !y  clri  = 0 for  all  i * k with  a . > 0 
ij  J aij  <J  U 

and  a.  . > 0 . 
kj 


Proof. 


The  proof  is  by  induction.  Since  xQ  is  an  extrame  point  the  lemma  is 
true  for  j = 0.  Suppose  it  is  true  for  j > 0.  If  Step  3 of  the  algorithm  applies 

°j+l  = Dj  and  there  is  nothing  to  prove.  In  the  case  that  Step  4 of  the 
algorithm  is  used  we  have 


Tj+  1 ~ lx  I x - 0 for  all  i # I with  a. . > 0 } 
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and 


c.  . , , - c.  for  i = i and  all  i with  a = 0 . 

i,J+l  U ij 

This  proves  the  first  part  of  the  lemma.  Furthermore,  c'  , C c . = 0 for 

i,j+l  k,j+l 

any  k * f with  a,  > 0 and  any  i with  a.  . = 0 because  c'  C c,  =0  and 
kj  ij  ij  kj 

c!  C c.  0.  The  last  statement  of  the  lemmas  follows  from  the  observation 
U t J 

that  a'  c..  = 0 for  any  i 1 l . 

tt. 

1J  -1 

Now  suppose  D^+  ^ is  determined  by  either  Step  S,  6 or  7 c.  Then  the  first 

part  of  the  lemma  follows  from  Lemma  1.  For  every  i * i with  a =0  and 

ij 

every  v with  a > 0 we  have 
vj 

c'  . a 

1,1+1  v , j+ 1 i,  J+ 1 v]  w 1,1+1  j 

The  first  term  on  the  right  hand  side  of  this  equality  is  zero  since 

c.  , t span  {c..  | all  i with  a..  =0}  , 
i,l+l  U 1 U ; 

the  second  term  is  zero  because,  by  Lemma  1,  c.  . . and  p,  are  conjuqate 

i,j+l  1 

with  respect  to  C . The  last  statement  of  the  lemma  is  obvious  since  p t T 

j 1 

If  Step  7a  applies,  then  x...  is  an  extreme  point  and  D-^ , has  the 

1+1  1+1 

required  properties.  Finally  assume  that  Step  7d  is  used.  Part  i)  of  the 

lemma  is  a consequence  of  Lemma  1 and  the  equality 

cL.  a 

,0  G.  = c'  C c.  - — p!  C c.  = 0 

f,l+l  i,i+l  fi  i,j+l  ^ . j i,j+l 

for  all  i with  a,  . - 0 . Furthermore, 

U * 

c'  .a. 

c!  , , , C c ■ , , = c.  . C c . - — ^ — — c!  . .Cp.-p  .c ! ,,  ,Cc.  . , . =0 

1 , J+  1 V,J+1  1,1+1  V]  W 1,]+1  1 KVJ  1,1+1  f,l+l 

for  every  i with  a,  , = 0 and  every  v^i  with  a >0.  The  last  statement 
ij  vj 

of  the  lemma  is  true  because 

a'  p.  = 0 and  a'  c.  , , = 0 
a,  . j a..  f,l+l 

ij  i 


for  every  i * l with  ». . > 0 

U 


-19- 


The  following  two  lemmas  give  conditions  under  which  is  a 

quasi-stationary  point. 

Lemma 

i)  If  cr ^ < a*  and  s is  determined  by  Step  la,  then  x.+  ^ is  a quasi- 
statjonary  point. 

ii)  If  a- . < a*  , x,  is  a quasi-stationary  point  and  s,  is  determined  by 
Step  lb,  then  XJ+ | is  a quasi-stationary  point. 


Proof. 


i)  It  suffices  to  show  that  g'  . c. . = 0 for  all  i with  a,  , =0.  We  have 

J+l  ij  U 

c'  g.,  . = c'  g.  - c'  . C s. 
kj  j+l  kj  j kj  j 

= c!  ,g.  - c'  C [ )]  ( 6.  .c'  ,g.)c  . 

kj  j kj  -1  i]  ij  ) ij 
a.  . =0 
IJ 

= c'  g.  - 0.  . c'  C c.  . c'  . g.  = 0 
kj  j kj  kj  kj  kj  j 

for  every  k with  a > 0 . 

ii)  Since  x.  is  a quasi-stationary  point 

c'  g =0  for  ail  i with  a.  . - 0 . 

ij  j i) 

Thus  s = c„  and  c ' . C c„.  = 0 for  all  i with  a. . = 0 implies 
j i)  ij  *J  i] 

c1  g , = 0 for  all  i with  a..  = 0 . 

ij  J+l  U 

Finally,  c^.  g.+1  = 0 since  cr.  = gjs./s^  C s.  . 

Lemma  4. 


i)  If  cr  < cr*,  then  x.  , is  a quasi-stationary  point 
j )’  j+l 


ii)  if  x is  not  a quasi-stationary  point,  then  either  x.+^  is  a quasi- 
stationary  point  or  there  are  more  constraints  active  at  x than 


there  are  at  x,  . 

J 


iii)  For  every  j at  least  one  of  the  points  x.,  x,+j,.  • • >xj+n  [ is  a quasi- 


stationary  point. 
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Proof. 


i)  If  a . < it*  , then  cannot  be  determined  by  Step  7e.  Indeed,  if  s 

is  determined  by  Step  7e,  then  v.  = -1  and  s'.  C s.  < 0.  Thus  <r.  = oo 

J J J “ J 

and  or  - cr*  . 

J I 

Since  xQ  is  a quasi-stationary  point,  it  follows  from  Lemma  3,  that 
the  statement  is  true  for  j = 0.  Now  assume  that  the  statement  is  true 
for  j-1.  Then  either  is  a quasi-stationary  point  or  s^  is  determined 
by  Step  la).  In  either  case  it  follows  from  Lemma  3,  that  x.+^  is  a 
quasi-stationary  point  provided  a.  < it*. 

ii)  If  x.  is  not  a quasi-stationary  point,  then  by  the  first  part  of  the 

lemma,  <r . . = <r*  Therefore,  s.  is  either  determined  by  Step  la  or 
J-1  J-1  J 

Step  7e.  In  either  case  all  constraints  that  are  active  at  x will  also 

j 

be  active  at  x and  the  statement  follows  from  part  i)  of  the  lemma. 

iii)  Since  every  extreme  point  of  {x  | Ax  < b}  is  a quasi-stationary  point 
the  last  statement  of  the  lemma  follows  immediately  from  part  ii). 

Theorem . 

The  algorithm  terminates  after  a finite  number  of  steps  with  a stationary 

point. 

Proof. 


Suppose  the  algorithm  terminates  with  x..  By  Step  1 of  the  algorithm 

either  <r.  . < <r*  . or  c!  ,g.  = 0 for  all  i with  a.  . = 0.  Thus  using  part  i) 
j-1  j-1  U J lj 

of  Lemma  4 we  see  that  x,  is  a quasi-stationary  point.  This  means  that 
there  are  numbers  X.  such  that 


l 


g = VQ(x.)  = ) \.  a 

o, . > 0 ij 

U 
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Since  by  Lemma  2,  and  Step  1 of  the  algorithm 


X.  = g'c,  . < c'g  <0  for  all  i with  a > 0 , 

1 J U - *J  i - ij 

the  Kuhn -Tucker-conditions  are  satisfied:  i.e.  , x.  is  a stationary  point. 

If  C is  positive  definite,  then  there  are  only  finitely  many  quasi- 
stationary points.  If  C is  not  positive  definite  then  there  can  be  infinitely 
many  quasi-stationary  points.  However,  there  are  only  finitely  many  with 
different  values  of  Q(x).  If  x^  is  any  quasi-stationary  point  such  that 

x . # x.,  then  Q(x.  ) < Q(x  ).  Therefore,  it  follows  from  part  iii)  of 
j+i  j j+i  ) 

Lemma  4,  that  the  algorithm  terminates  after  a finite  number  of  steps. 

Remark. 

A numerical  study  involving  the  method  given  in  this  paper  and  several 
other  algorithms  for  quadratic  minimization  problems  is  currently  undertaken 
at  the  University  of  Waterloo.  The  results  will  be  reported  elsewhere. 
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